<!--
========================================================================
 *  Copyright by KNIME AG, Zurich, Switzerland
 *  Website: http://www.knime.com; Email: contact@knime.com
 *
 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License, Version 3, as
 *  published by the Free Software Foundation.
 *
 *  This program is distributed in the hope that it will be useful, but
 *  WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 *  GNU General Public License for more details.
 *
 *  You should have received a copy of the GNU General Public License
 *  along with this program; if not, see <http://www.gnu.org/licenses>.
 *
 *  Additional permission under GNU GPL version 3 section 7:
 *
 *  KNIME interoperates with ECLIPSE solely via ECLIPSE's plug-in APIs.
 *  Hence, KNIME and ECLIPSE are both independent programs and are not
 *  derived from each other. Should, however, the interpretation of the
 *  GNU GPL Version 3 ("License") under any applicable laws result in
 *  KNIME and ECLIPSE being a combined program, KNIME AG herewith grants
 *  you the additional permission to use and propagate KNIME together with
 *  ECLIPSE with only the license terms in place for ECLIPSE applying to
 *  ECLIPSE and the GNU GPL Version 3 applying for KNIME, provided the
 *  license terms of ECLIPSE themselves allow for the respective use and
 *  propagation of ECLIPSE together with KNIME.
 *
 *  Additional permission relating to nodes for KNIME that extend the Node
 *  Extension (and in particular that are based on subclasses of NodeModel,
 *  NodeDialog, and NodeView) and that only interoperate with KNIME through
 *  standard APIs ("Nodes"):
 *  Nodes are deemed to be separate and independent programs and to not be
 *  covered works.  Notwithstanding anything to the contrary in the
 *  License, the License does not apply to Nodes, you are not required to
 *  license Nodes under the License, and you are granted a license to
 *  prepare and propagate Nodes, in each case even if such Nodes are
 *  propagated with or for interoperation with KNIME.  The owner of a Node
 *  may freely choose the license terms applicable to such Node, including
 *  when such Node is propagated with or for interoperation with KNIME.
====================================================================
-->
<body>
Contains the implementation of a node which reads ARFF files. ARFF files are
ASCII data files. They consist of a header, specifying the structure and format
of the data, and a data section which contains the actual data vectors.
<br>
There is a special format "sparce ARFF files" which is not supported yet.
<p>The ARFFTable (see <code>ARFFTable.java</code>) contains a static method
to create a <code>DataTableSpec</code> from an ARFF file. With this table spec
an ARFFTable can then be instantiated to read the data from the file.<br>
The actual reading of the data from the file is done in the <code>ARFFRowIterator</code>
(see <code>ARFFRowIterator.java</code>).</p>
<p>The following is a specification of the ARFF format. It is taken from the
WEKA project pages. Please see their project home page <a
  href="http://www.cs.waikato.ac.nz/~ml/"> http://www.cs.waikato.ac.nz/~ml/</a>
to check for their copyright and license.
<h2>Attribute-Relation File Format (ARFF)</h2>

<p>April 1st, 2002</p>

<p>An ARFF (Attribute-Relation File Format) file is an ASCII text file that
describes a list of instances sharing a set of attributes. ARFF files were
developed by the Machine Learning Project at the Department of Computer Science
of The University of Waikato for use with the <a
  href="http://www.cs.waikato.ac.nz/~ml/">Weka machine learning software</a>.
This document descibes the version of ARFF used with Weka versions 3.2 to 3.3;
this is an extension of the ARFF format as described in the data mining book
written by Ian H. Witten and Eibe Frank (the new additions are string
attributes, date attributes, and sparse instances).</p>

<p>This explanation was cobbled together by Gordon Paynter
(gordon.paynter@ucr.edu) from the Weka 2.1 ARFF description, email from Len
Trigg (lenbok@myrealbox.com) and Eibe Frank (eibe@cs.waikato.ac.nz), and some
datasets. It has been edited by Richard Kirkby (rkirkby@cs.waikato.ac.nz).
Contact Len if you're interested in seeing the ARFF 3 proposal.</p>

<h2>Overview</h2>

<p>ARFF files have two distinct sections. The first section is the <b>Header</b>
information, which is followed the <b>Data</b> information.</p>

<p>The <b>Header</b> of the ARFF file contains the name of the relation, a
list of the attributes (the columns in the data), and their types. An example
header on the standard IRIS dataset looks like this:</p>

<pre>
   % 1. Title: Iris Plants Database
   % 
   % 2. Sources:
   %      (a) Creator: R.A. Fisher
   %      (b) Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
   %      (c) Date: July, 1988
   % 
   &#064;RELATION iris

   &#064;ATTRIBUTE sepallength  NUMERIC
   &#064;ATTRIBUTE sepalwidth   NUMERIC
   &#064;ATTRIBUTE petallength  NUMERIC
   &#064;ATTRIBUTE petalwidth   NUMERIC
   &#064;ATTRIBUTE class        {Iris-setosa,Iris-versicolor,Iris-virginica}
  </pre>

<p>The <b>Data</b> of the ARFF file looks like the following:</p>

<pre>
   &#064;DATA
   5.1,3.5,1.4,0.2,Iris-setosa
   4.9,3.0,1.4,0.2,Iris-setosa
   4.7,3.2,1.3,0.2,Iris-setosa
   4.6,3.1,1.5,0.2,Iris-setosa
   5.0,3.6,1.4,0.2,Iris-setosa
   5.4,3.9,1.7,0.4,Iris-setosa
   4.6,3.4,1.4,0.3,Iris-setosa
   5.0,3.4,1.5,0.2,Iris-setosa
   4.4,2.9,1.4,0.2,Iris-setosa
   4.9,3.1,1.5,0.1,Iris-setosa
  </pre>

<p>Lines that begin with a % are comments. The <b>&#064;RELATION</b>, <b>&#064;ATTRIBUTE</b>
and <b>&#064;DATA</b> declarations are case insensitive.</p>

<h2>Examples</h2>

<p>Several well-known machine learning datasets are distributed with Weka in
the $WEKAHOME/data directory as ARFF files.</p>

<h2>The ARFF Header Section</h2>

<p>The ARFF Header section of the file contains the relation declaration and
attribute declarations.</p>

<h3>The &#064;relation Declaration</h3>

<p>The relation name is defined as the first line in the ARFF file. The
format is:</p>

<pre>
    &#064;relation &lt;relation-name&gt;
   </pre>
<p>where &lt;relation-name&gt; is a string. The string must be quoted if the
name includes spaces.</p>

<h3>The &#064;attribute Declarations</h3>

<p>Attribute declarations take the form of an orderd sequence of <b>&#064;attribute</b>
statements. Each attribute in the data set has its own <b>&#064;attribute</b>
statement which uniquely defines the name of that attribute and it's data type.
The order the attributes are declared indicates the column position in the data
section of the file. For example, if an attribute is the third one declared then
Weka expects that all that attributes values will be found in the third comma
delimited column.</p>

<p>The format for the <b>&#064;attribute</b> statement is:</p>

<pre>
    &#064;attribute &lt;attribute-name&gt; &lt;datatype&gt;
   </pre>
<p>where the <i>&lt;attribute-name&gt;</i> must start with an alphabetic
character. If spaces are to be included in the name then the entire name must be
quoted.</p>

<p>The <i>&lt;datatype&gt;</i> can be any of the four types currently
(version 3.2.1) supported by Weka:</p>
<ul>
  <li>numeric</li>

  <li>&lt;nominal-specification&gt;</li>
  <li>string</li>
  <li>date [&lt;date-format&gt;]</li>
</ul>
<p>where &lt;nominal-specification&gt; and &lt;date-format&gt; are defined
below. The keywords <b>numeric</b>, <b>string</b> and <b>date</b> are case
insensitive.</p>

<h4>Numeric attributes</h4>

<p>Numeric attributes can be real or integer numbers.</p>

<h4>Nominal attributes</h4>

<p>Nominal values are defined by providing an &lt;nominal-specification&gt;
listing the possible values: {&lt;nominal-name1&gt;, &lt;nominal-name2&gt;,
&lt;nominal-name3&gt;, ...}</p>

<p>For example, the class value of the Iris dataset can be defined as
follows:</p>
<pre>

    &#064;ATTRIBUTE class        {Iris-setosa,Iris-versicolor,Iris-virginica}
   </pre>

<p>Values that contain spaces must be quoted.</p>

<h4>String attributes</h4>

<p>String attributes allow us to create attributes containing arbitrary
textual values. This is very useful in text-mining applications, as we can
create datasets with string attributes, then write Weka Filters to manipulate
strings (like StringToWordVectorFilter). String attributes are declared as
follows:</p>

<pre>
    &#064;ATTRIBUTE LCC    string
   </pre>

<h4>Date attributes</h4>

<p>Date attribute declarations take the form:</p>

<pre>

    &#064;attribute &lt;name&gt; date [&lt;date-format&gt;]
   </pre>
<p>where &lt;name&gt; is the name for the attribute and &lt;date-format&gt;
is an optional string specifying how date values should be parsed and printed
(this is the same format used by SimpleDateFormat). The default format string
accepts the ISO-8601 combined date and time format:
&quot;yyyy-MM-dd'T'HH:mm:ss&quot;.</p>

<p>Dates must be specified in the data section as the corresponding string
representations of the date/time (see example below).</p>

<h2>ARFF Data Section</h2>

<p>The ARFF Data section of the file contains the data declaration line and
the actual instance lines.</p>

<h3>The &#064;data Declaration</h3>

<p>The <b>&#064;data</b> declaration is a single line denoting the start of
the data segment in the file. The format is:</p>
<pre>
    &#064;data
   </pre>

<h3>The instance data</h3>

<p>Each instance is represented on a single line, with carriage returns
denoting the end of the instance.</p>

<p>Attribute values for each instance are delimited by commas. They must
appear in the order that they were declared in the header section (i.e. the data
corresponding to the nth <b>&#064;attribute</b> declaration is always the nth
field of the attribute).</p>

<p>Missing values are represented by a single question mark, as in:</p>

<pre>
    &#064;data
    4.4,?,1.5,?,Iris-setosa
   </pre>

<p>Values of string and nominal attributes are case sensitive, and any that
contain space must be quoted, as follows:</p>
<pre>
    &#064;relation LCCvsLCSH

    &#064;attribute LCC string
    &#064;attribute LCSH string

    &#064;data
    AG5,   'Encyclopedias and dictionaries.;Twentieth century.'
    AS262, 'Science -- Soviet Union -- History.'
    AE5,   'Encyclopedias and dictionaries.'
    AS281, 'Astronomy, Assyro-Babylonian.;Moon -- Phases.'
    AS281, 'Astronomy, Assyro-Babylonian.;Moon -- Tables.'
   </pre>

<p>Dates must be specified in the data section using the string
representation specified in the attribute declaration. For example:</p>
<pre>
    &#064;RELATION Timestamps

    &#064;ATTRIBUTE timestamp DATE &quot;yyyy-MM-dd HH:mm:ss&quot; 

    &#064;DATA 
    &quot;2001-04-03 12:12:12&quot;

    &quot;2001-05-03 12:59:55&quot;
   </pre>

<h2>Sparse ARFF files</h2>

<p>Sparse ARFF files are very similar to ARFF files, but data with value 0
are not be explicitly represented.</p>

<p>Sparse ARFF files have the same header (i.e <b>&#064;relation</b> and <b>&#064;attribute</b>
tags) but the data section is different. Instead of representing each value in
order, like this:</p>
<pre>
    &#064;data
    0, X, 0, Y, &quot;class A&quot;
    0, 0, W, 0, &quot;class B&quot;

   </pre>
<p>the non-zero attributes are explicitly identified by attribute number and
their value stated, like this:</p>
<pre>
    &#064;data
    {1 X, 3 Y, 4 &quot;class A&quot;}
    {2 W, 4 &quot;class B&quot;}
   </pre>

<p>Each instance is surrounded by curly braces, and the format for each
entry is: &lt;index&gt; &lt;space&gt; &lt;value&gt; where index is the attribute
index (starting from 0).</p>

<p>Note that the omitted values in a sparse instance are <b>0</b>, they are
not &quot;missing&quot; values! If a value is unknown, you must explicitly
represent it with a question mark (?).</p>

<p><b>Warning</b>: There is a known problem saving SparseInstance objects
from datasets that have string attributes. In Weka, string and nominal data
values are stored as numbers; these numbers act as indexes into an array of
possible attribute values (this is very efficient). However, the first string
value is assigned index 0: this means that, internally, this value is stored as
a 0. When a SparseInstance is written, string instances with internal value 0
are not output, so their string value is lost (and when the arff file is read
again, the default value 0 is the index of a different string value, so the
attribute value appears to change). To get around this problem, add a dummy
string value at index 0 that is never used whenever you declare string
attributes that are likely to be used in SparseInstance objects and saved as
Sparse ARFF files.</p>
</body>
